Source color modification on a digital nonlinear editing system

ABSTRACT

A system and method for generating a representation of a color modification to be applied to segments on a digital nonlinear editing system, where each segment is a component of a media composition, and represents a section of a digital media. An indication of a modification to be applied to a color attribute of a segment is received, and the source from which the segment originates is identified. The indication of the color modification is then stored, and, as a result, the color modification is applied to other segments that originate from the identified source. Source color modification is applied to a section of a digital media on a digital nonlinear editing system. a media segment represents the section of a digital media. The segment is a component of a media composition, and originates from a source data structure. The source data structure also represents the section of the digital media. The section is received, and the first source data structure from which the segment originates is identified. It is determined whether the first source data structure includes a color modification attribute, where the color modification attribute defines a color modification to be applied to sections of the digital media represented by segments that originate from the source data structure. If the source data structure includes the color modification attribute, the color modification is applied to the section of media.

BACKGROUND

Digital non-linear editing (DNLE) is a process by which digital mediamay be edited. DNLE, as the name implies, is performed on digital mediastored as data in digital media files on a digital random access medium.DNLE may be conducted in a non-linear fashion because the digital mediafiles in which the digital media is stored can be randomly accessed.Thus an editor may access a piece of the digital media without having toproceed sequentially through other pieces of the digital media stored inthe same or other digital media files. More than one editor also may beable to access different pieces of the same digital mediacontemporaneously. The digital media may be a digitized version of afilm or videotape or digital media produced through live capture onto adisk of a graphics or animation software application. Example commercialDNLE systems include the Media Composer® or the Avid® Symphony™ videoproduction system or NewsCutter® news editing system available from AvidTechnology, Inc. For a more detailed description of DNLE, see DigitalNonlinear Editing, New Approaches to Editing Film and Video, 1993, byThomas Ohanian.

Color modification is a class of operations that may be performed bothto correct color errors due to process errors and to adjust the colorsused in the video for artistic expression. Such color modifications mayinclude enhancing contrasts or color in an image to give a program anoverall “look,” or applying special effects to selected segments. Othercolor modifications may be made by an editor during an editing sessionto correct problems with color or lighting resulting from the source ofthe media. Such corrections may include color balancing for camera andlighting differences, correcting for film processing differences,matching colors and tones from shot to shot, or adjusting video levelsfor differences in source tapes, source decks, etc.

Digital images are comprised of an array of picture elements calledpixels. For a given image, color modifications may be applied to allpixels in the image or pixels comprising a portion of the image. Indigital video signal processing, a variety of data formats can be usedto represent the color of pixelx within a digital image. Formats may beclassified into two major categories: composite signals and componentsignals. Component formats represent a color as multiple components,each component defining a value along a dimension of the color space inwhich the color being represented is defined. A composite video is ananalog signal that uses a high frequency subcarrier to encode colorinformation. The subcarrier is a sinewave of which the amplitude ismodulated by the saturation of the color represented by the signal, andthe hue of the color is encoded as a phase difference from a colorburst. Analog composite signals generally are used to broadcasttelevision video signals.

There are a variety of component formats used to represent color. RGB(Red, Green, Blue) format represents a color with a red component, agreen component and a blue component. CMY (Cyan, Magenta, Yellow, Black)format represents a color with a cyan component, a magenta component,and a yellow component. CMY is a format commonly used by printers. TheCMY components are color opposites of RGB components. In athree-dimensional coordinate system, each component of either the RGB orthe CMY format represents a value along an axis, the combination of thevalues defining a cubic color space.

The data formats HSL (Hue, Saturation, Lightness or Luminance) and HSV(Hue, Saturation, Value) represent a color with a hue component, asaturation component, and a luma component. In a three-dimensionalcoordinate system, the luma component represents a value along a lumaaxis, the hue component represents the angle of a chroma vector withrespect to the luma axis and the saturation component represents themagnitude of the chroma vector. The combination of the values defines ahexagonal cone-shaped color space around the luma axis.

YCrCb, YUV, and YIQ are three formats that represent a color with a lumacomponent Y, and two chroma components, Cr and Cb, U and V, or I and Q,respectively, that define a chroma vector. In a three-dimensionalcoordinate system, each component of either the YCrCb, YUV, and YIQformat represents a value along an axis, the combination of the valuesdefining a cylindrical color space around the luma axis. The chromacomponents define the chroma vector. In data formats with a lumacomponent, the luma component can be used independently to represent apixel in a black and white image to be displayed, for example, with ablack and white monitor.

A typical color modification in HSL color space may include increasing acolor component or a combination of color components for all pixels ineach digital image of a section of digital media. Typically, an editoraccesses a segment of a composition that represents the section of mediathrough an editor interface and inputs desired color modificationsthrough the editor interface. Some systems permit an editor to applycolor modifications to only portions of a digital image. Portions of adigital image can also be described as one or more pixels. For example,an editor may select with a mouse, keyboard, or some other editor inputdevice a portion of the image and define color modifications for theselected portion. A suitable commercial system for color modification isAvid Media Illusion™ available from Avid Technology, Inc. The Avid MediaIllusion Reference Guide, available from Avid Technology, Inc. is hereinincorporated by reference. Other commercial software applications may beused.

SUMMARY

One problem with correct techniques for color modification is that acolor modification generally cannot be specified for several segments ina composition originating from a common source. An editor generallyaccesses every segment derived from a source individually to make thecolor modification. This process can be time-consuming and prone toerror. An editor may intend on making the same modification to a firstand second segment originating from a common source, but themodifications may be inconsistent. Different editors also may be workingon the different segments of the same source. Although color matchingfrom one segment to the other is one solution to this inconsistency, theother segment is not always available, and having to color match addsmore time to the editing process.

Source color modification permits a color modification to be specifiedfor several segments originating from a common source. Such source colormodification may be combined with other color modifications madegenerally to a program or part of a program.

Accordingly, in one aspect, a method generates a representation of acolor modification to be applied to segments on a digital nonlinearediting system, where each segment is a component of a mediacomposition, and represents a section of a digital media. An indicationof a modification to be applied to a color attribute of a segment isreceived, and the source from which the segment originates isidentified. The indication of the color modification is then stored,and, as a result, the color modification is applied to other segmentsthat originate from the identified source. In one embodiment, the colormodification is stored in a source data structure that represents thesource of the segment. In another embodiment, the color modification isstored in the other segments originating from the source.

In yet another embodiment, a source relationship attribute of thesegment is accessed to identify the source. The source relationshipattribute indicates a source of the segment, and is used to determine acommon source from which the segment and other segments on the systemoriginate. Consequently, the source of the first segment is identifiedin accordance with the source relationship attribute. In anotherembodiment, the source relationship attribute is received.

In yet another embodiment, the segment includes a source identifier thatis accessed to determine the source of the segment.

In another aspect, a method applies color modification to a section ofdigital media on a digital nonlinear editing system. A media segmentrepresents the section of a digital media. The segment is a component ofa media composition, and originates from a source data structure. Thesource data structure also represents the section of the digital media.The section is received, and the first source data structure from whichthe segment originates is identified. It is determined whether the firstsource data structure includes a color modification attribute, where thecolor modification attribute defines a color modification to be appliedto sections of the digital media represented by segments that originatefrom the source data structure. If the source data structure includesthe color modification attribute, the color modification is applied tothe section of media.

In another embodiment, it is determined whether the segment includesanother color modification defined by the composition that to be appliedto the section of the digital media represented by the segment. If themedia composition includes the other color modification, the other colormodification is applied to the first section of the digital media.

In another aspect, provided is method of performing natural colormatching. An indication of a selected destination color component isreceived. The destination color having a first luminance and a sourcedestination color component having a second luminance. A product of aratio of the first and second luminance and the value of selecteddestination color component is determined. The values of the selecteddestination color component are adjusted according to the determinedproduct.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings,

FIG. 1 is a block diagram illustrating relationships between source datastructures and source media;

FIG. 2 a is a block diagram illustrating relationships betweencomposition data structures in a composition;

FIG. 2 b is a block diagram illustrating parallel and serialrelationships between the composition data structures of FIG. 2 b;

FIG. 3 a is a diagram illustrating a user interface for a digitalnonlinear editing system;

FIG. 3 b is a diagram illustrating a section of media;

FIG. 4 is a block diagram illustrating relationships between the sourcedata structures of FIG. 2 and the composition data structures of FIG. 3a;

FIG. 5 is a table illustrating an embodiment of a relationship;

FIG. 6 is a block diagram illustrating an embodiment of effects ofsource color modification and composition color modification on segmentswithin a sequence;

FIG. 7 is a block diagram illustrating an embodiment of source datastructures and composition data structures;

FIG. 8 a is a data flow diagram illustrating an embodiment of a sourcecolor modifier;

FIG. 8 b is a data flow diagram illustrating an embodiment of acomposition color modifier;

FIG. 9 is a flow chart illustrating an embodiment of an editing processimplementing composition color modification and a source colormodification;

FIG. 10 is a flow chart illustrating an embodiment of a process ofidentifying a source data structure defined by a source relationshipattribute;

FIG. 11 is a flow chart illustrating an embodiment of a process ofidentifying a composition data structure defined by a compositionrelationship attribute;

FIG. 12 is a data flow diagram illustrating an embodiment of a mediaplayer implementing source color modification and composition colormodification;

FIG. 13 is a flow chart illustrating an embodiment of a process ofimplementing source color modification and composition colormodification during playback;

FIG. 14 is an example embodiment of a user interface for colormodification;

FIG. 15 is an illustration describing operation in the HSL color space;and

FIG. 16 is a block diagram illustrating an embodiment of the storage ofsource and composition color modifications.

DETAILED DESCRIPTION

The following detailed description should be read in conjunction withthe attached drawing in which similar reference numbers indicate similarstructures. All references cited herein are hereby expresslyincorporated by reference.

A DNLE may track or have access to information that indicates howsegments of media in a composition may be related by the source of themedia data that they represent. This information may be used to applycolor modification to multiple segments at the same time. Suchmodification is referred to herein as source color modification.

The relationships between segments of a source media in a compositionare represented by references to source data structures representingmedia that can be combined or composed into a multi-media composition.Color modifications, including source color modifications, are generallydefined within the context of a composition. Compositions arerepresented by composition data structures or components. Datastructures for source media and compositions are described in moredetail below.

Relationships between segments of media in a composition, the datastructures that represent these relationships, and the relationshipsbetween these data structures themselves are described in one or moreU.S. patents, including U.S. Pat. No. 5,752,029 by Michael J. Wissnerand issued May 12, 1998, entitled Method And Apparatus For RepresentingAnd Editing Multimedia Compositions Using References To Tracks In TheComposition To Define Components Of The Composition (The Wissnerpatent), U.S. Pat. No. 5,267,351, filed on Dec. 22, 1989 by Stephen J.Reber et al. entitled MEDIA STORAGE AND RETRIEVAL SYSTEM (the Reberpatent), U.S. patent application Ser. No. 09/054,761 filed Apr. 3, 1998by Eric C. Peters entitled Computer System And Process For TransferringMultiple High Bandwidth Streams Of Data Between Multiple Storage UnitsAnd Multiple Applications In A Scalable And Reliable Manner (the Petersapplication), incorporated herein by reference. Source relationships anddata structures also are described in the OMF Interchange® Specification(OMF), version 2.1, 1997, available from the OMF Developers' Desk ofAvid Technologies, Inc., and available on the Internet at theURL:http://www.avid.com/3rdparty/omfi, and in the Advanced AuthoringFormat specification (AAF), herein incorporated by reference.

A general summary of such source relationships and data structures thatmay be used will now be described.

A DNLE system typically permits an editor to create a multimediacomposition. A multimedia composition is collection of relationshipsbetween time-varying media data, representing how the data should besynchronized and combined over time. Time-varying data, may be, forexample, video or audio data, but is not limited to such data. Staticdata that does not vary with time, for example, still pictures and text,is a subset of time-varying data, and may also be included in amultimedia composition. The data are related by grouping them intodifferent types of components, the combination of which forms acomposition. A method and apparatus for representing such a mediacomposition is described in one or more U.S. patents, including theWissner patent, incorporated herein by reference.

Media data used in a composition includes digitized versions of physicalsource media such as video or audio tape, compact disk, computergenerated images, etc. Physical source media are also referred to hereinas physical media. Digitized versions of physical media available foruse are referred to herein as digital media sources or digital sources.Digital sources may include digital samples of physical media or mayhave been created from application software for graphics, animation, orword processing, etc. A digital source created by application softwareis referred to herein as a digitally created source. A digital sourcedigitally sampled directly from a physical media is herein referred toas an original digital media source or an original digital source. Adigital source may represent a single image or a single sample, and maybe a copy of or a portion of another digital source. Digital sources arestored in digital media files.

Representations of digital sources and representations of physical mediaare referred to herein as source data structures. Source data structuresmay be stored in data files in a data base or any other format. A sourcedata structure representing a digital source may be stored in the samemedia data file as the digital source it represents or in a separatedata file.

A source data structure includes information describing the media,whether digital or physical, that it represents. The information mayinclude: how the digital source was created; an identification of thecorresponding physical media; the sample rate (and therefore theduration of a sample), and the length of each sample in bytes; anindication of the section of the physical media that it represents andthe time offset from the source physical media of its first sample. Theunits of this offset is the sample duration for the digital source.

Multiple digital sources of the same physical media and theircorresponding source data structures also may be stored if desired.Storing multiple digital sources and source data structures allows thecomposition to support the interchange of media at different levels ofvisual or audio quality for different purposes. For example, one digitalsource might have a level of quality which is suitable for output tovideo tape, whereas an alternative digital source might be useful fordisplaying in a small window on a computer screen. Examples of such asystem for storing and accessing multiple digital sources of a singlephysical media are described in the Reber patent and the Petersapplication, incorporated herein by reference. A commercial storagesystem suitable for storing media data files and composition dataincludes the MediaShare® storage system available from AvidTechnologies, Inc. Other commercial systems may be used.

A source data structure called a source clip represents a singletime-contiguous section of media. As with other source data structures,a source clip does not include the actual media data of the digitalsource, but only references it, for example by referring to a data file.A source clip represents a digital source, which could be an originaldigital source or a digitally created digital source. A source clip thatrepresents an original digital source or a digitally created source isreferred to herein as a master source clip or a master clip. A sourceclip that represents a digital source that is a section or a copy of anoriginal digital source or a digitally created source is herein referredto as a subclip. A source data structure that represents a physicalmedia is herein referred to as a physical source media object orphysical media object. Source clips are described in OMF and AAF,incorporated herein by reference.

A source clip may include a source identifier that identifies anothersource data structure that represent either a digital source or physicalmedia that includes the section of media represented by the source clip.If the source clip represents a digitally created digital source, thesource identifier is a null value because the digitally created digitalsource does not originate from another source. A source clip that doesnot represent a digitally created digital source also includes a sourceoffset. The source offset identifies a starting position relative to thesection of media represented by the source data structure from which thesource clip originates.

If a first source data structure is referred to as originating from asecond source data structure herein, the second source data structure isthe source of the first source data structure. If a first source datastructure is referred to as indirectly originating from a second sourcedata structure herein, the second source data structure is the source ofat least a third source data structure that is the source of the firstsource data structure. “At least” a third data structure means thatthere may be multiple source data structures between the first andsecond source data structures, each data structure originating from theother in a source chain between the first and second source datastructures. For example, a subclip may indirectly originate from aphysical media object, as described below in connection with FIG. 1.

An example illustration of offset is a source clip that has an offset of40 units and represents an original digital source. The source clip thusoriginates from a data structure that represents a physical media, asdiscussed in more detail below. If the data structure from which thesource clip originates includes an offset of 100 and represents units100-200 of a physical source, the 40 units defined by the source clipare offset from unit 100 of the physical source. The source thusrepresents a section of media beginning at unit 40 of the digitalsource, and which ends at unit 140 of the physical media.

A source data structure to which a source clip refers also may refer toanother source data structure, which also may refer to yet anothersource data structure, etc. This type of multiple layering is describedin the Wissner patent and by OMF, herein incorporated by reference.

An example embodiment of such a multilayered representation isillustrated in FIG. 1. The physical media object 48 is a source datastructure that presents the physical source media 54 from which originaldigital media sources 50 are created. Each original digital source 50may be stored in digital media file as described in the Reber patent.Each original digital source 50 is a digitized portion of the physicalmedia 54, where any of the original digital sources 55 could be adigitization of the entire physical media 54. For each original digitalsource 50 created from the physical media 54, a master source clip 46that represents the original digital source is also created.

Other digital sources such as 52 may be specified as sections of theoriginal digital sources 50. Alternatively, other digital sources may becreated as copies of an original digital source 5. For each otherdigital source 52 created, subclips 44 are created to represent thedigital source 52. Each subclip 44 originates from a master clip 46which may originate from a physical media object 48. Consequently, eachsubclip 44 may indirectly originate from a physical media object 48.

In order to support the editing of compositions of a variety of media,composition data structures are used for organizing and storinginformation concerning a composition and operations manipulating thosecomposition data structures. The basic building blocks of a compositionare called components. A composition is structured as a tree ofcomponents including a root component. A component may or may not havesubcomponents, depending on its type. A component may be considered afunction over time because it includes information for producing thestate of its portion of the composition at any time within its range. Acomponent thus represents a time-dependent sequence of media data orsources called a media stream.

The composition data structures used for representing the components ofa composition exclude the media data itself. The composition datastructures include indications of or references to the media data andrepresentations of the relationships between and combinations of themedia data which form the composition. Thus, compositions are storedseparately from the media data to which they refer, and allow manycompositions to use the same media data without duplicating it.

Several types of data structures may be combined to form a composition.FIG. 2 a illustrates one possible combination of components. Eachsegment 6 represents a section of media as a component of a composition.Although each segment also may represent more complex structures such asa sequence or a track group, which is discussed further below, a segmentas referred to herein represents a single time-contiguous section ofmedia in the context of a composition, unless otherwise specified. Asegment includes a source identifier that identifies a source clip.

Transitions 8 are components which are located between two segments in asequence of components, and indicate how a presentation shouldtransition from displaying one segment to displaying the next segment.

A sequence 4 represents the serialization or concatenation in time of acollection of components. A sequence may define an ordered list ofsegments separated by transitions, and may itself be a segment withinanother sequence. The order of segments in a sequence defines the orderof interpretation or “playback.” Each sequence 4 may include a list ofsubcomponents and includes the subcomponent identifier of eachsubcomponent. An example of a playback system suitable for playing acomposition is described in the Wissner Patent and the Petersapplication and U.S. Pat. No. 5,045,940, filed Dec. 22, 1989 by Eric C.Peters entitled VIDEO/AUDIO TRANSMISSION SYSTEM AND METHOD (the Peterspatent), incorporated herein by reference. Also, a commercial playbacksystem may be used for playing compositions that implements the MediaEngine video playback system available from Avid Technology, Inc. thatis incorporated in the Avid® AirPlay® MP playback server system. MediaComposer from Avid Technology, Inc is a suitable commercial system forplayback as well as editing. Other commercial systems may be used.

A track group defines a parallel relationship between sequences orsegments, which are defined by tracks within the track group. Forexample, one track within a track group may be an audio sequence orsegment to be synchronized with a video sequence. Another track withinthe track group may be the video segment or sequence, and a third trackwithin a track group may be a background video effect segment orsequence to be combined with the video segment or sequence. Acomposition is essentially a track group wherein the parallel tracksbegin at a same point in time and end at a same point in time.

FIG. 2 b illustrates a representation of a composition 2 from which astill image is shown in FIG. 2 a, wherein sequences 4 and 5 arerepresented by parallel tracks 14 and 15, respectively. Each track 4 and5 may be considered a logical player channel, and therefore is of asingle media type. Because tracks 14 and 15 are subcomponents of acomposition 2, the sequences 4 and 5 start at the same point in time andend at the same point in time.

FIG. 4 illustrates the relationship between source data structures and amedia composition. The composition 2 includes 5 tracks, where each trackis a sequence, including a sequence 4 that includes several segments andtransitions. Sequence 4 includes a segment 6 that originates from asubclip 44. The subclip 44 originates from master clip 46 thatoriginates from the physical media object 48. Source clip 44 is thesource of segment 7 as well as segment 6. Segment 7 is a subcomponent ofsequence 5. Thus, the subclip 44 is the source of two segments 6 and 7which belong to two different sequences 4 and 5, respectively. Althoughnot shown in FIG. 4, the subclip 44 could be the source of a segmentused in an entirely different composition. The same kind of relationshipbetween source data structures and composition data structures can betrue at the master clip level also. A master source clip or a physicalmedia object can be the source of multiple subclips linked to segmentsof multiple compositions.

Having now described the various kinds of source relationships and anembodiment of the data structures representing these relationships andhow they are used to define a composition, an embodiment of source colormodification will now be described.

Source color modification provides a simple, time-saving method ofapplying a single color modification to several segments originatingfrom a common source. The common source may be defined by a relationshipsource relationship attribute that may be defined by an editor.

FIG. 5 illustrates a table 60 that lists possible source relationshipsfor source color modification and possible composition relationships forcomposition color modification, which will be discussed in more detailbelow. The table 60 includes a source relationship column 62 and acomposition relationship column 64.

The source relationship attribute can be defined as a segmentrelationship 61, a source clip relationship 63, a master cliprelationship 65 or an physical media relationship 67. If the sourcerelationship is defined as the segment relationship 61, the colormodification is applied only to the active segment during playback. Anactive segment, as used herein, means either the segment currently beingedited on an editing system or a segment currently being played on aplayback system. If the source relationship is defined as the sourceclip relationship 61, during playback the color modification is appliedto any segment, used within any composition on the system, thatoriginates from the same source clip as the active segment. If thesource relationship attribute is defined as a master clip relationship,during playback the color modification is applied to any segment thatdirectly or indirectly originates from the same master clip from whichthe active segment indirectly originates. If the source relationshipattribute is defined to be the physical media relationship, the colormodification is applied to any segment that indirectly originates fromthe same physical media object from which the active segment indirectlyoriginates.

For example, an editor may define the source relationship to be a sourceclip relationship. If an editor notices that a segment representing filmfootage is too dark, the editor can increase the luminance of thesegment to therefore brighten the image. During playback, themodification increase in luminance may be applied to any segmentoriginating from the source clip.

Composition color modification provides a method for an editor to applya single color modification to several segments within a composition. Acomposition relationship attribute defines the ancestor of the activesegment that is used to determine a common ancestor to which alldescendants in the composition have in the composition structure colormodification applied during playback.

Referring to the table 60 of FIG. 6, the composition relationshipattribute may be defined to be a segment relationship 69, a selectedsegments relationship 71, a sequence relationship 73, or a compositionrelationship 75. If the composition relationship attribute is defined tobe the segment relationship 69, only the active segment has the colormodification applied during playback. If the composition relationshipattribute is defined to be the selected segments relationship 71, onlythe selected segments have the color modification applied duringplayback. If the composition relationship attribute is defined to be thesequence relationship 73, all segments descended from or included inthat sequence have the color modification applied during playback. Ifthe composition relationship attribute is the composition relationship75, all segments of the composition have the color modification appliedduring playback.

The combination of composition color modification and source colormodification applied to a single segment will be described withreference to FIG. 6. Suppose that various footage has been shot by afilm crew to be used in a movie. Included in this footage is some fromscene A, footage from another scene B, footage from another scene C, andfootage from yet another scene D. The film editors decide to use digitalnon-linear editing to combine the footage, make color modifications, andadd special effects, etc. to produce the finished movie.

First, the editors create original digital sources from the filmfootage. Thus, an original digital source A, an original digital sourceB, an original digital source C, and an original digital source D arecreated. As discussed above, for each digital source created, a mastersource clip that represents the digital source is created also. Thus, amaster source clip A, a master source clip B, a master source clip C anda master source clip D are created.

The editors then decide that the footage of scene B can be divided intotwo separate shots: shot 1 and shot 2. Thus, from the original digitalsource B, a digital source of shot 1 and a digital source of shot 2 arecreated. In response to the creation of the two new digital sources, asubclip of shot 2 is created to represent the digital source of shot 2,and a subclip of shot 1 is created to represent the digital source ofshot 1.

The editors then decide to combine the footage of the various scenestogether. Using a digital non-linear editor, the editors create acomposition that includes a sequence 66 that includes a segment 74originating the master clip A, a segment 76 that originates from thesubclip of shot 2, a segment 78 that originates from the master clip C,a segment 80 that originates from the subclip of shot 1, and a segment82 that originates from the master clip D.

While viewing the various segments in the video display, an editors maynotice that the segment of shot 2 is too dark. The editors may decide tomake a color modification to increase the luminance of this segment.

Using source color modification, the editors define the master cliprelationship as the source relationship attribute. The editors thenincrease the luminance of the segment of shot 2 by 5 IRE. Because thesource relationship attribute was set to master clip relationship, uponplayback both the segment of shot 2 and segment of shot 1, which bothindirectly originate from the master clip B, have luminance increased by5 IRE as shown in item 68 of FIG. 6.

The editor may then realize that all the segments of the sequence 66 aretoo bright. The editor may set the composition relationship attribute toa sequence relationship. The editor may then decrease the luminance ofthe active segment by 2 IRE. Because the composition relationshipattribute is defined as a sequence relationship, all segments within thesequence 66 have their luminance decreased by two IRE during playback,as shown by row 70 in FIG. 7. Row 72 shows the combined colormodification of both the composition color modification and the sourcecolor modification that is applied to each segment within the sequence66 during playback.

FIG. 7 illustrates an embodiment of source and composition datastructures that can be used for source and composition colormodification. A segment 130 includes information 132 about the sectionof media that the segment 130 represents. This information may include asource identifier 132, a parent identifier 134, a source colormodification attribute 138, and a composition color modificationattribute 140. The source and composition color modification attributes138 and 140 may themselves be partitioned into a plurality of colormodification attributes.

The source identifier 134 identifies the source data structure fromwhich the segment 130 originates. The parent identifier 136 identifiesthe parent of the segment in a composition 158. The source colormodification attribute 138 represents a source color modification to beapplied to the segment 130 during playback. The composition colormodification attribute 140 defines a composition color modification tobe applied to the section of media represented by the segment 130 duringplayback. It should be noted that if a source or color modification isnot defined for a data structure, a color modification attribute may bea null value, or the color modification attribute may not be included ina data structure.

In FIG. 7, the source identifier 134 points to a subclip data structure122. The subclip data structure 122 includes information 124 about thesection of media that the subclip represents. The information 124includes a source identifier 126 and may include a source colormodification attribute 128, depending upon whether the source colormodification was defined for the subclip 122 during an editing session.The source identifier 126 of the subclip 122 refers to a master clipdata structure 114. The master clip 114 includes information 116 aboutthe digital media that the master clip 114 represents. The information116 may include a source identifier 118 and may include a colormodification attribute 120. If the master clip 114 represents adigitally created digital source, the source identifier 118 may be anull value indicating that the master clip does not originate fromanother source.

The source identifier 118 of the master clip 114 may refer to a physicalmedia object 106. The physical media object 106 includes information 108that describes the physical media that the physical media object 106represents. The information 108 may include a source identifier 112 andmay include a color modification attribute 110. The information 108includes a source identifier 112 if the physical media that the physicalmedia object represents was created from another physical media. Forexample, the physical media may be a video tape that was created from afilm through a telecine process. The source identifier 112 refers to thephysical media object that represents the film.

In the example of FIG. 7, the parent identifier 134 of the segment datastructure 130 refers to a first sequence data structure 142. The firstsequence 142 includes information 144 about the first sequence 142. Thefirst sequence information 144 may include a parent identifier 146 andincludes a subcomponent identifier 148. The first sequence information144 would not include a parent identifier 146 if the sequence 142 isitself a composition, which is not shown in FIG. 8. The first sequenceinformation 144 may include an ordered list of subcomponents of thefirst sequence 142, where the order of the subcomponents determines theorder in which the subcomponents are played during playback. For eachsubcomponent, the information 144 may include a subcomponent identifier148.

In the illustration of FIG. 7, the subcomponent identifier 148identifies segment 130. The parent identifier 146 identifies a secondsequence 150 of which the first sequence 142 is a subcomponent. Thesecond sequence 150 includes information 152 about the sequence 150. Thesecond sequence information 152 includes information analogous to thatdescribed for a second sequence information 152 includes a parentidentifier 154 that refers to the composition data structure 158. Thesecond sequence information 152 would not include a parent identifier154 if the second sequence 150 is itself a composition which is notshown in FIG. 8. The information 152 also includes an ordered list ofsubcomponents, the order of the subcomponents determining the order ofplayback of the subcomponents during playback. For each subcomponent,the information 152 includes a subcomponent identifier 156. Thesubcomponent identifier 156 identifies first sequence 142 as asubcomponent of second sequence 150. The parent identifier 154identifies the composition 158 as the parent of sequence 150.

Composition 158 includes information 160 about the composition 158. Thecomposition information 160 includes the sequences of each track of thecomposition, wherein the sequences are subcomponents of the composition158. For each subcomponent of the composition 158, the information 162includes a subcomponent identifier 162. The subcomponent identifier 162identifies sequence 150 as a subcomponent of composition 158.

FIG. 8A is a dataflow diagram that illustrates how the source datastructures illustrated in FIG. 7 can be used to implement source colormodification during an editing session. a source color modifier 82receives user defined color modifications to an active segment 84 and asource relationship attribute 86. Using the color modification 84 andthe attribute 86, the source color modifier 82 accesses the source datastructures 88 and modifies the source data structures 88 in accordancewith the color modification 84 and the attribute 86.

FIG. 9 is a flowchart illustrating an editing process implementingsource color modification 186 and composition color modification 188.With reference to FIG. 9, the implementation of source colormodification 186 by the source color modifier 82 during an editingsession will now be described. First, in step 164, an indication of acolor modification is received. Next, in step 166, an editor maydetermine whether the color modification is a source color modificationor a composition color modification. Such a determination is made, forexample, by accessing an indication of the color modification modeselected by an editor. If it is determined that source colormodification shall be applied, at the next step 178, the source colormodifier determines the source relationship to be used to determine acommon source of the active segment and other segments on the system.Determining the source relationship to be used can be accomplished byeither accessing a default source relationship attribute or by receivinga source relationship attribute defined by an editor.

Next, in step 180, the source color modifier identifies a source datastructure of the active segment defined by the source relationship. FIG.10 is a flowchart that defines in more detail the step 180 ofidentifying the source data structure. First, in step 190, the sourcecolor modifier 82 determines whether the current source data structureis the source data structure defined by the source relationship. Thefirst time through the loop defined by steps 190-194, the current sourcedata structure is the active segment. If the current source datastructure is the defined source data structure, then the step ofidentifying the source data structure of the active segment is done, asillustrated in step 196. If the current source data structure is not thedefined source data structure, in the next step 192, then the sourcecolor modifier 82 reads the source identifier of the current source datastructure. In step 194, the source color modifier then accesses thesource data structure identified by the source identifier. Steps 190-194are repeated until the source data structure defined by the sourcerelationship attribute is identified.

Returning to the flowchart of FIG. 9, in the next step 182, the sourcecolor modifier stores an indication of the source color modification inthe source data structure identified in step 180.

In an alternative embodiment of source color modification, colormodifications may be stored as an entry of a composition source table,as opposed to being stored in a source data structure. Each entry in thecomposition source table include two pieces of data: the identificationof a source data structure for which a source color modification of asegment within the composition is defined, and an attribute representingthe source color modification. Each time a source color modification isdefined for a segment within the composition, an entry may be stored inthe source table identifying the source data structure from which thesegment originates, which is identified by the source relationshipattribute, and the attribute representing the color modification.

The composition source table allows a source color modification to beshielded from other compositions on the system that use material fromthe defined source. The source modification may be defined specificallyfor segments of the same composition that originate from a commonsource, but not for all segments on the system originating from thecommon source. Thus, other editors are not forced to implement colormodifications on the composition that they are editing that may havebeen defined by a different editor working on a different composition.

Another benefit of using the source composition table is that itprovides an editor with the option of not applying a source modificationto the composition. An editor may select whether the composition sourcetable is to be activated. For example, if a colorist defines a series ofsource color modifications for a composition, the source data structuresand the modifications are stored in entries of the composition sourcetable as described above. At a later point in time, an editor may accessthe composition, but not activate the table, and thus not apply themodifications defined therein.

For an active segment during playback, the color modification system mayfirst determine if there is a source color modification defined for theactive segment data structure. If a color modification is defined forsegment data structure, the modification may be applied. If nomodification is defined in the active segment data structure, thecomposition source table may be accessed. If it is determined that thereis an entry for the subclip from which the active segment originates,the source color modification defined for the subclip is applied to theactive segment. If it is determined that no entry for the subclip ispresent in the table, it is determined whether the table includes anentry for the master clip from which the segment originates. If it isdetermined that there is an entry for the master clip from which theactive segment originates, the source color modification defined for themaster clip is applied to the active segment. If it is determined thatno entry for the master clip is present in the table, it is determinedwhether the table includes an entry for the physical media object fromwhich the segment originates. If it is determined that there is an entryfor the physical media object from which the active segment originates,the source color modification defined for the physical media object isapplied to the active segment.

FIG. 8B is a dataflow diagram illustrating how composition colormodification can be implemented by a composition color modifier 90. Thecomposition color modifier 90 receives color modifications 92 to beapplied to the active segment. The composition color modifier 90 thenaccesses a default relationship attribute or receives a compositionrelationship attribute 94 defined by an editor. The composition colormodifier then accesses the composition data structures 96 and modifiesthe composition data structures 96 in accordance with the colormodification 92 and the relationship attribute.

Referring again to FIG. 10, an implementation of composition colormodification implemented by the composition color modifier 90 will nowbe described. As described above with respect to source colormodification, the first step is step 164 of receiving an indication of acolor modification. The digital media editor then determines whethersource color modification or composition color modification is to beapplied in step 166.

If composition color modification is to be applied, in the next step168, the composition color modifier determines the compositionrelationship to be used. The composition relationship is defined byeither a default composition relationship attribute or a compositionrelationship attribute defined by an editor.

In the next step 170, the composition color modifier identifies theancestor data structure of the active segment defined by the compositionrelationship attribute.

FIG. 11 provides a more detailed flowchart of step 170. First, in step190, the composition color modifier determines whether the currentcomposition data structure is the ancestor data structure defined by thecomposition relationship attribute. The first time through the loopdefined by steps 198-202, the current composition data structure is theactive segment. If the current composition data structure is the definedancestor data structure, then the step 170 of identifying the ancestordata structure is done, as illustrated in step 204. If the currentcomposition data structure is not the defined ancestor, the next step200 reads the parent identifier of the current composition datastructure. In the next step 202, the composition color modifier accessesthe composition data structure identified by the parent identifier.Steps 198-202 are then repeated until the ancestor data structuredefined by the composition relationship attribute is identified.

Returning to FIG. 9, in the step 172, the composition color modifierdetermines the descendants of the identified ancestor data structure,for example, by accessing the subcomponent list of the identified datastructure and accessing the subcomponent identifiers for eachsubcomponent, and traversing downward in the composition from theidentified ancestor data structure. In step 174, for each identifieddescendant or subcomponent of the identified ancestor data structure, anindication of the color modification is stored in a descendant datastructure. Traversing a composition to perform editing operations oncomponents is described in the Wissner patent.

Another aspect of source and composition color modification is theapplication of the color modifications to segments within a compositionduring playback. FIG. 12 is a dataflow diagram that illustrates animplementation of source and composition color modification duringplayback. A digital media player 98 receives an indication of a startingpoint 100 within a composition. The starting point 100 may be thebeginning of the composition by default or may be defined by input froman editor on a user interface. The digital media player 98 accessescomposition data structures 96 to properly synchronize the playing ofsections of a digital media. For each segment identified within thecomposition, the digital media player identifies source data structures88 from which the active segment originates, and determine whether anysource color modification attributes are defined for these source datastructures 88. The digital media player also uses the source datastructures 88 to determine the digital sources from which the digitalmedia should be accessed. The digital media player also determineswhether the active segment has a composition color modification. Thedigital media player then applies the source color modification and thecomposition color modification to the active segment. This process isrepeated for every segment of the composition, or every segment includedin a portion of the composition selected by the user, such that asequence of digital images 102 is produced as defined by thecomposition. A playback system may apply color modification sequentiallyon a segment, one digital image at a time. More specifically, a playbacksystem may apply color modification sequentially on an digital image.Such a system is described in a U.S. patent application entitled“Digital Nonlinear Editing System Implementing Primary and SecondaryColor Correction”, filed Apr. 15, 1999 by Rob Gonsalves and RayCacciatore.

FIG. 13 is a flowchart that illustrates an implementation of source andcomposition color modification during playback. In step 206, the digitalmedia player 98 accesses the next segment in a composition. Next, instep 208, the digital media player 98 determines whether a source colormodification attribute is defined for this data structure. If there is asource color modification attribute defined for this data structure, thedigital media player determines whether there is a temporarily storedsource color modification attribute for this active segment.

It should be noted that the first time through the loop defined by steps208-216, the source data structure is the active segment. For thisreason, the first time through the loop defined by steps 208-216, atstep 210 there is not a temporarily stored color modification attributefor the active segment. Alternatively, for the first time through theloop defined by steps 208-216, step 210 may be skipped. The datastructure that holds the temporarily stored source color modificationattribute for the active segment also may be initialized between steps206 and 208.

Returning to the flowchart of FIG. 13, if there is not a temporarilystored source color modification attribute, in step 212 the digitalmedia player temporarily stores the source color modification attributefor the active segment. If the digital media player determines in step210 that there is already a temporarily stored source color modificationattribute for the active segment or determines in step 208 that there isnot a color modification attribute defined for this data structure, thedigital media player proceeds to step 214. In step 214, the digitalmedia player determines whether there is a source data structure fromwhich the current source data structure originates.

If the media player determines that there is a source data structurefrom which the current source data structure originates, the mediaplayer proceeds to step 216. In step 216, the media player accesses thesource data structure from which the current source data structureoriginates. Steps 208-216 are repeated until there is not a source datastructure from which the current source data structure originates.

If there is not a source data structure from which the current sourcedata structure originates, the digital media player proceeds to step218. In step 218, the digital media player determines whether there is atemporarily stored source color modification attribute for the activesegment. If there is a temporarily stored source color modification forthis segment, the digital media player in step 220 applies thetemporarily stored source color modification attribute to the activesegment.

After applying the temporarily stored source color modificationattribute to the active segment, or after determining that there is nota temporarily stored source color modification attribute for thissegment, the digital media player proceeds to step 222. In step 222, thedigital media player determines whether there is a composition colormodification defined for the active segment. If there is a compositioncolor modification defined for this segment, the digital media playerproceeds to step 224 where it applies the composition color modificationto the active segment.

After applying the composition color modification to the active segmentor after determining that there is not a composition color modificationto find for this active segment, the digital media player then proceedsto step 206 where it accesses the next segment in the composition. Steps206-224 are repeated until the entire composition or the portion of thecomposition defined by a user has been played.

FIG. 14 is an example embodiment of a user interface 400 for colormodification. A three-image display 270 includes a current image display229, a previous image display 228, and a next image display 230. Thecurrent image display 229 represents an active segment of a sequence,and the previous image display 228 and the next image display 230represent the previous and next segments of the sequence, respectively.A control panel, for example the control panel 272 of the current image229, may be associated with each image display. For the current imagedisplay, for example, the control panel may include a previous segmentbutton 232, a next segment button 234, a previous unmodified segmentbutton 236, a next unmodified segment button 238, a play button 240, anda stop button 242. Each image may also include a scroll bar, for examplescroll bar 261 of the current image 229. Each scroll bar may include aposition indicator 262 which indicates the temporal position of thecurrent image 229 within the segment that it represents.

The three image display 270 allows the editor to see the effects ofsource and composition color modification. If pressed, the previoussegment button 232 replaces the active segment with the previous segmentin the current image display 229. In the example of FIG. 14, if theprevious segment button 232 is pressed, a digital image from theprevious segment appears in the current image display 229. Analogously,if the next segment button 234 is pressed, a digital image from the nextsegment appears in the current image display 229.

During an editing session, various source and program modifications maybe defined. Depending on the source and composition relationshipattributes chosen during the editing session, various segments of thesequence being represented by the three image display 270 may beaffected. The previous unmodified segment button 236 and the nextunmodified segment button 238 allow an editor to view the effects ofsource and composition color modification in an efficient manner.

If the previous unmodified segment button 236 is pressed, a previoussegment closest in time to the active segment that has not yet had colormodification applied replaces the active segment in the current imagedisplay 229. The unmodified segment buttons allow an editor to quicklydetermine which segments of a sequence have not been color modified.Rather than having to use the next and previous buttons 232 and 234repeatedly to traverse all the segments in a sequence, even when thesegments have already been modified, an editor can simply press buttons236 and 238 and to view the closest unmodified segments.

In an example embodiment, color function buttons allow an editor toselect function screens for both source and composition colormodification. For example, tab 244 allows a user to select colorfunctions for source color modification, and tab 246 allows a user toselect functions for composition color modification. The color functionscreens allow an editor to perform specific color modificationfunctions. Color function buttons may include an HSL button 254, achannels button 252, a levels button 250, and a curves button 248. TheHSL button 254 brings up an HSL function screen that allows a user todefine changes to pixel colors in HSL color space, including definingcolor modifications to pixel colors as a function of the luma of apixel. Channels button 252 allows a user to access a channels screen,where channels in this context refers to the red, green, and bluecomponents of an RGB component format. The RGB screen allows a user tomodify values of the RGB components as a function a component the RGBcomponents or combinations of the RGB components.

The levels button 250 allows a user to access a levels screen in which auser can determine the effects of RGB color modifications on the luma ofthe pixel, and to alter the luma of the pixel as a function of the lumaof the pixel. Example embodiments of the color modifications availableon a levels screen is described in U.S. patent application “Apparatusand Method for Generating and Displaying Histograms of Color Images forColor Correction,” by Robert Gonsalves, filed even herewith.

The curves button 248 allows a user to access a curves screen. Anexample embodiment of a curves screen is illustrated in user interface400, in which the curves button has been selected. The curves screenallows a user to define color modifications for a red, green, or bluecomponent of a color, or for all three components of the color. Thecurve screen includes a red graph 280, a green graph 282, and a bluegraph 284 for defining the functions of the individual components, and amaster graph 286 for defining a function that is applied to all threecolor components. In each graph, the horizontal axis represents an inputvalue of the function, while the vertical axis represents the outputvalue of the function. Each graph may include control points that ifadded, moved or deleted alter a curve representing the function, therebyaltering the function.

For example, for the green graph 282, by altering any of the controlpoints 292, 294, 296, and 298, the green curve 300 is altered, therebyredefining a function for the green component. The new values for thefunctions are determined using interpolation, for example, linearinterpolation, cubic splines, or Bezier curves.

In an example embodiment, the curve screen of the user interface 400allows a user to color match using colors from images of the three-imagedisplay 270, text input from a text entry field such as 288, or othercolor sources. For example, a user may select a color from the imagedisplayed in the next image display screen 230 and apply it to the imagein the current image display screen 229. The curve screen allows theuser to preview the effect of adding a color to an image, using thegraphs 280, 282, 284, and 286. The curve screen also provides a colormatching gadget 257 to assist an editor in color matching. The colormatching gadget 257 includes an input swatch 261 and an output swatch260. The RGB values of the selected colors for color matching aredisplayed in the swatches. The RGB graphs 280, 282, 284, and 286 allowthe user to preview these colors without committing to a change in thefunctions represented by these graphs.

Typically, an editor picks from the current image display 229 for thesource color, the value of which is displayed in the input color swatch261, and picks from a known good color, the value of which is displayedin the output color swatch 260. A known good color may be provided by acolor palette, or from another image, such as the image displayed in theprevious image display 228 or the next image display 230. The user mayselect how they would like to match the color. There may be eightchoices, including: master, R+G+B, R+G, G+B, B+G, Red, Green, Blue. Ifan editor picks R+G+B as the match type, as displayed in selection box258 of the user interface 400, and hits the match color button 256, thena new control point is added to each of the red, green, and blue graphs280, 282, and 284, respectively. Hash marks or some other indicator onthe graphs represent the changes to those graphs resulting from thecolor match. For example, the hash mark 294′ on the green graph 282represents a change in the value of the control point 294, which altersthe green component function.

In an embodiment of color matching, the hue and saturation of the outputcolor are automatically adjusted to match how objects behave undernatural lighting environments. For example, if an object is illuminatedfrom natural light, the ratios between the RGB values of the objectremain proportional from areas of the object highly illuminated by thelight to darker areas of the object not illuminated by as much light.

Natural color matching operates in the following manner. Given a sourcecolor, R_(S), G_(S), B_(S), for example, the color represented in theinput color swatch 286, and a destination color R_(D), G_(D), B_(D), forexample, the output color represented by the output color swatch 260, anadjusted destination color, R′_(D), G′_(D), B′_(D) is determined.

The luminance of the source color, Y_(S) may be defined as:Y _(S)=0.299·R _(S)+0.587·G _(S)+0.114·B _(S)  Equation 1

Input red/luminance_(ratio, ρRS), input green/luminance ratio, ρ_(GS),and input blue/luminance ratio, ρ_(BS), may be defined as:$\begin{matrix}{{Equation}\quad 2\text{:}} & \quad \\{\rho_{RS} = \frac{R_{S}}{Y_{S}}} & \quad \\{{Equation}\quad 3\text{:}} & \quad \\{\rho_{GS} = \frac{G_{S}}{Y_{S}}} & \quad \\{{Equation}\quad 4\text{:}} & \quad \\{\rho_{BS} = \frac{B_{S}}{Y_{S}}} & \quad\end{matrix}$

The luminance of the destination color, Y_(D), may be defined as:Y _(D)=0.299·R _(D)+0.587·G _(D)+0.114·B _(D)  Equation 5:

Output red/luminance ratio, ρ_(RD), output green/luminance ratio,ρ_(GD), and output blue/luminance ratio, ρ_(BD), may be defined as:$\begin{matrix}{{Equation}\quad 6\text{:}} \\{{\rho_{RD} = \frac{R_{D}}{Y_{S}}}\quad} \\{{Equation}\quad 7\text{:}} \\{{\rho_{GD} = \frac{G_{D}}{Y_{D}}}\quad} \\{{Equation}\quad 8\text{:}} \\{\rho_{BD} = \frac{B_{D}}{Y_{D}}}\end{matrix}$

Adjusted red/luminance ratio, ρ_(′RD), adjusted green/luminance ratio,ρ′_(GD), and adjusted blue/luminance ratio, ρ′_(BD), may be defined as:$\begin{matrix}{{Equation}\quad 9\text{:}} \\{\rho_{D}^{\prime} = {{\frac{R_{D}}{Y_{S}}\quad\rho_{\quad{RS}}} = {\frac{\quad R_{\quad S}}{\quad Y_{\quad S}}R_{D}}}} \\{{Equation}\quad 10\text{:}} \\{\rho_{D}^{\prime} = {{\frac{G_{D}}{Y_{D}}\quad\rho_{GS}} = {\frac{G_{S}}{Y_{S}}G_{D}}}} \\{{Equation}\quad 11\text{:}} \\{\rho_{D}^{\prime} = {{\frac{B_{D}}{Y_{D}}\quad\rho_{BS}} = {\frac{B_{S}}{Y_{S}}B_{D}}}}\end{matrix}$

In a color difference space, such as YCbCr, the Y value represents theluminance and the two color components, CbCr, represent the chrominance.Chrominance may be defined by two values: hue, which is defined by theangle of the vector (0,0)-->(Cb,Cr), and saturation which is themagnitude of this vector and may be defined as √{square root over (C_(b)²+C_(r) ²)}.

FIG. 15 illustrates the effect of natural color matching in HSL space.By matching the ratios of RGB to luminance, a resulting modified imageadopts the hue of the specified destination color, H_(D), but maintainsthe luminance Y_(S) of the source color. If a destination vector isdrawn from the black point to the destination color, the adjusted color,C′_(D), is located at the intersection of this destination vector and aplane defined by the source luma Y_(S). The saturation of the adjustedcolor, C′_(D), is the magnitude of an adjusted vector that may bedefined as √{square root over (C′_(b) ²+C′_(r) ²)}.

Natural color matching may be used to match any combination of the red,green, and blue components. R+G+B matching was described above. Naturalmatching with the master curve affects luminance only. Six othercombinations that can be used are: R+G, G+B, B+G, Red, Green, and Blue.

An embodiment of natural color matching allows a user to select theother color match combinations, R+G, G+B, B+G, Red, Green, Blue. Theprocess of naturally matching colors for these selections is similar tomatching with all three RGB components as described above. Ratios aredetermined by dividing the selected components of a color by theweighted sum of the components not selected. One may consider twoclasses of these selections: single component selections including Red,Green, and Blue, and two component selections including R+G, G+B, andB+G.

For single component selections, given the source color components,R_(S), G_(S), and B_(S), and the destination color components R_(D),G_(D), B_(D), an adjusted destination component may be determined asfollows, using an adjusted red component, R′_(D), as an example. Aweighted sum of the green and blue source components, G_(D) and B_(D)approximates source luminance, Y_(S). This weighted sum may be definedby the following equation: $\begin{matrix}{{Equation}\quad 12\text{:}} & \quad \\{Y_{S} = \frac{{0.587 \cdot G_{S}} + {0.114 \cdot B_{S}}}{0.587 + 0.114}} & \quad\end{matrix}$

The source red/luminance ratio, ρ_(RS), of the source color isdetermined. The source red/luminance ratio ρ_(RS) may be defined by thefollowing equation: $\begin{matrix}{{Equation}\quad 13\text{:}} & \quad \\{\rho_{RS} = \frac{R_{S}}{Y_{S}}} & \quad\end{matrix}$

A weighted sum of the green and red destination components, G_(S) andB_(S), may approximate the destination luminance, Y_(D). This weightedsum may be defined by the equation: $\begin{matrix}{{Equation}\quad 14\text{:}} & \quad \\{Y_{D} = \frac{{0.587 \cdot G_{D}} + {0.114 \cdot B_{D}}}{0.587 + 0.114}} & \quad\end{matrix}$

The destination red/luminance ratio, ρ_(RS), is determined. The sourcered/luminance ratio ρ_(RS) may be defined by the following equation:$\begin{matrix}{{Equation}\quad 15\text{:}} & \quad \\{\rho_{RD} = \frac{R_{D}}{Y_{D}}} & \quad\end{matrix}$

The adjusted red component, R′_(D), may be determined by combiningEquations 12-15 to produce the following equation: $\begin{matrix}{{Equation}\quad 16\text{:}} \\{R_{D}^{\prime} = {{\frac{\rho_{RD}}{\rho_{RS}}\quad R_{S}} = {\frac{G_{S}}{Y_{D}}R_{D}}}}\end{matrix}$

For two component selections, given the source color components, R_(S),G_(S), and B_(S), and the destination color components R_(D), G_(D),B_(D), an adjusted destination component may be determined as follows,using adjusted red and blue components, R′_(D and B′D) as examples. Thegreen source component approximates source luminance, Y_(S).G _(S) =Y _(S)  Equation 17:

A source red/luminance ratio ρ_(RS) and a source blue/luminance ratioρ_(BS) is determined. These ratios may be defined by the followingequations: $\begin{matrix}{{Equation}\quad 18\text{:}} & \quad \\{{\rho_{RS} = \frac{R_{S}}{Y_{S}}}\begin{matrix}{{Equation}\quad 19\text{:}} & \quad \\{\rho_{BS} = \frac{B_{S}}{Y_{S}}} & \quad\end{matrix}} & \quad\end{matrix}$

The green component of the source color, G_(D), may approximateluminance, Y_(D)Y _(D) =G _(D)  Equation 20:

The destination red/luminance ratio ρ_(RD) and the destinationblue/luminance ratio ρ_(BD) are determined. These ratios may be definedby the following equations: $\begin{matrix}{{Equation}\quad 21\text{:}} \\{{\rho_{RD} = \frac{R_{D}}{Y_{D}}}\begin{matrix}{{Equation}\quad 22\text{:}} & \quad \\{\rho_{BD} = \frac{B_{D}}{Y_{D}}} & \quad\end{matrix}}\end{matrix}$

Equations 17-22 may be combined to determine the adjusted red and bluecomponents R′_(D) and B′_(D) as defined by the following equations:$\begin{matrix}{{Equation}\quad 23\text{:}} & \quad \\{R_{D}^{\prime} = {{\frac{\rho_{RD}}{\rho_{RS}}\quad R_{S}} = {\frac{Y_{S}}{Y_{D}}R_{D}}}} & \quad \\{{Equation}\quad 24\text{:}} & \quad \\{B_{D}^{\prime} = {{\frac{\rho_{BD}}{\rho_{BS}}\quad B_{S}} = {\frac{Y_{S}}{Y_{D}}B_{D}}}} & \quad\end{matrix}$

Thus, natural color matching adjusts the values of the selecteddestination color components as a product of the ratio of the source anddestination luminance of the selected components and the value ofdestination component.

FIG. 16 is a block diagram illustrating an embodiment of source andcomposition color modifications data structures 138 and 140,respectively. Both the source and composition modification structures138 and 140 may include color modifications specific to a HSL, channels,levels, and curves color functions and parameters. Parameters may beconsidered a subclass of functions as parameters are essentially simplelinear functions or constants. Both the data structures 138 and 140 mayinclude references to other data structures specific to the colorfunctions, for example, HSL modifications data structure 404, channelsmodifications data structure 406, levels modifications data structure408, and curves modifications data structure 410 These functions mayhave been defined through the user interface 400 described above inconnection with FIG. 14, using the color function buttons 254, 252, 250,and 248 to access the HSL, channel, levels, and curves screen,respectively.

In FIG. 16, if there are multiple sub-structures that are identical inform, then only one of the sub-structures are illustrated in fulldetail. Items in a data structure followed by a “□” character indicatesa sub-structure indicated by an arrow to the sub-structure. For example,in the source color modification data structure 138, the □ next to “HSL”indicates that there is a sub-structure 404 defining HSL colormodifications. Items followed by a “•” character indicates a parameter,with a category in parentheses. For example, in the channels colormodification data structure 406, the • next to “preview_mode (num)”indicates that there is a numeric parameter defining the preview modeselected by a user. Items followed by a “ƒ” indicate a function.Functions may be stored as fixed length array of input/output pairs ofvalues. These values are used to calculate a lookup table that definethe function. For example, in the curves color modification datastructure 410, the ƒ next to “red (rgb)” indicates a function for RGBcomponent curves. Using functions and lookup tables to define colormodifications is described in U.S. patent application entitled“Multi-tone Representation of a Digital Image on a Digital Non-LinearEditing System,” by Robert Gonsalves, filed even herewith, and U.S.patent application entitled “Color Modification on a Digital Non-LinearEditing System,” (the Cacciatore application) by Raymond D. Cacciatoreand Robert Gonsalves, filed even herewith. The data structures of FIG.16 may be used to define coefficients for a matrix or values of a lookuptable as described in the Cacciatore application.

A multimedia composition may be represented and edited with a typicalcomputer system. It should be understood that the invention is notlimited to any specific computer described herein. Many other differentmachines may be used to implement source color modification. Such asuitable computer system includes a processing unit which performs avariety of functions and a manner well-known in the art in response toinstructions provided from an application program. The processing unitfunctions according to a program known as the operating system, of whichmany types are known in the art. The steps of an application program aretypically provided in random access memory (RAM) in machine-readableform because programs are typically stored on a non-volatile memory,such as a hard disk or floppy disk. When a user selects an applicationprogram, it is loaded from the hard disk to the RAM, and the processingunit proceeds through the sequence of instructions of the applicationprogram.

The computer system also includes a user input/output (I/O) interface.The user interface typically includes a display apparatus (not shown),such as a cathode-ray-tube (CRT) display in an input device (not shown),such as a keyboard or mouse. a variety of other known input and outputdevices may be used, such as speech generation and recognition units,audio output devices, etc.

The computer system also includes a video and audio data I/O subsystem.Such a subsystem is well-known in the art and the present invention isnot limited to the specific subsystem described herein. The audioportion of subsystem includes an analog-to-digital (A/D) converter (notshown), which receives analog audio information and converts it todigital information. The digital information may be compressed usingknown compression systems, for storage on the hard disk to use atanother time. a typical video portion of subsystem includes a videoimage compressor/decompressor (not shown) of which many are known in theart. Such compressor/decompressors convert analog video information intocompressed digital information. The compressed digital information maybe stored on hard disk for use at a later time. An example of such acompressor/decompressor is described in U.S. Pat. No. 5,355,450.

It should be understood that one or more output devices may be connectedto a playback system or editing system implementing source and/orcomposition color modification. Example output devices include a cathoderay tube (CRT) display, liquid crystal displays (LCD) and other videooutput devices, printers, communication devices such as a modem, storagedevices such as disk or tape, and audio output. It should also beunderstood that one or more input devices may be connected to theediting or playback system. Example input devices include a keyboard,keypad, track ball, mouse, pen and tablet, communication device, anddata input devices such as audio and video capture devices and sensors.It should be understood that source and color modification are notlimited to the particular input or output devices used in combinationwith the computer system or to those described herein.

The editing or playback system may be a general purpose computer systemwhich is programmable using a computer programming language, such as“C++,” JAVA or other language, such as a scripting language or evenassembly language. The computer system may also be specially programmed,special purpose hardware. In a general purpose computer system, theprocessor is typically a commercially available processor, such as theseries ×86 and Pentium processors, available from Intel, similar devicesfrom AMD and Cyrix, the 680×0 series microprocessors available fromMotorola, and the PowerPC microprocessor from IBM. Many other processorsare available. Such a microprocessor executes a program called anoperating system, of which WindowsNT, Windows95 or 98, UNIX, Linux, DOS,VMS, MacOS and OS8 are examples, which controls the execution of othercomputer programs and provides scheduling, debugging, input/outputcontrol, accounting, compilation, storage assignment, data managementand memory management, and communication control and related services.The processor and operating system define a computer platform for whichapplication programs in high-level programming languages are written.

A memory system typically includes a computer readable and writeablenonvolatile recording medium, of which a magnetic disk, a flash memoryand tape are examples. The disk may be removable, known as a floppydisk, or permanent, known as a hard drive. A disk has a number of tracksin which signals are stored, typically in binary form, i.e., a forminterpreted as a sequence of one and zeros. Such signals may define anapplication program to be executed by the microprocessor, or informationstored on the disk to be processed by the application program.Typically, in operation, the processor causes data to be read from thenonvolatile recording medium into an integrated circuit memory element,which is typically a volatile, random access memory such as a dynamicrandom access memory (DRAM) or static memory (SRAM). The integratedcircuit memory element allows for faster access to the information bythe processor than does the disk. The processor generally manipulatesthe data within the integrated circuit memory and then copies the datato the disk after processing is completed. A variety of mechanisms areknown for managing data movement between the disk and the integratedcircuit memory element, and the invention is not limited thereto. Itshould also be understood that the invention is not limited to aparticular memory system.

Such a system may be implemented in software or hardware or firmware, ora combination of the three. The various elements of the system, eitherindividually or in combination may be implemented as a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a computer processor. Various steps of the process may beperformed by a computer processor executing a program tangibly embodiedon a computer-readable medium to perform functions by operating on inputand generating output. Computer programming languages suitable forimplementing such a system include procedural programming languages,object-oriented programming languages, and combinations of the two.

It should be understood that the playback system or editing system usedto implement source or composition modification is not limited to aparticular computer platform, particular processor, or particularprogramming language. Additionally, the computer system may be a multiprocessor computer system or may include multiple computers connectedover a computer network. It should be understood that each step of FIGS.9-11, and 13 may be separate modules of a computer program, or may beseparate computer programs. Such modules may be operable on separatecomputers.

Having now described some embodiments, it should be apparent to thoseskilled in the art that the foregoing is merely illustrative and notlimiting, having been presented by way of example only. Numerousmodifications and other embodiments are within the scope of one ofordinary skill in the art and are contemplated as falling within thescope of the invention.

1. A computer-implemented method of processing a color modification tobe applied to a plurality of segments of digital media, wherein eachsegment of the plurality of segments represents at least a section of adigital media source, the method comprising: receiving an indication ofa color modification to be applied to a first segment; receiving anindication of a relationship of the first segment with one or more ofthe other segments; storing an indication of the color modification suchthat the color modification is associated with at least one othersegment having the indicated relationship with the first segment.
 2. Acomputer-implemented method of processing a color modification to beapplied to a plurality of segments of digital media, wherein eachsegment of the plurality of segments represents at least a section of adigital media source, the method comprising: receiving an indication ofa color modification to be applied to a first segment; receiving anindication of a relationship of the first segment with one or more ofthe other segments; applying the color modification to at least oneother segment having the indicated relationship with the first segment.3. Apparatus for processing a color modification to be applied to aplurality of segments of digital media, wherein each segment of theplurality of segments represents at least a section of a digital mediasource, the system comprising: means for receiving an indication of acolor modification to be applied to a first segment; means for receivingan indication of a relationship of the first segment with one or more ofthe other segments; means for storing an indication of the colormodification such that the color modification is associated with atleast one other segment having the indicated relationship with the firstsegment.
 4. Apparatus for processing a color modification to be appliedto a plurality of segments of digital media, wherein each segment of theplurality of segments represents at least a section of a digital mediasource, the system comprising: means for receiving an indication of acolor modification to be applied to a first segment; means for receivingan indication of a relationship of the first segment with one or more ofthe other segments; means for applying the color modification to atleast one other segment having the indicated relationship with the firstsegment.
 5. A computer program product, comprising: a computer-readablemedium; computer program instructions stored on the computer-readablemedium that, when processed by a computer, instruct the computer toperform a method for processing a modification to be applied to aplurality of segments of digital media, wherein each segment of theplurality of segments represents at least a section of a digital mediasource, the method comprising: receiving an indication of a colormodification to be applied to a first segment; receiving an indicationof a relationship of the first segment with one or more of the othersegments; storing an indication of the color modification such that thecolor modification is associated with at least one other segment havingthe indicated relationship with the first segment.
 6. A computer programproduct, comprising: a computer-readable medium; computer programinstructions stored on the computer-readable medium that, when processedby a computer, instruct the computer to perform a method for processinga color modification to be applied to a plurality of segments of digitalmedia, wherein each segment of the plurality of segments represents atleast a section of a digital media source, the method comprising:receiving an indication of a color modification to be applied to a firstsegment; receiving an indication of a relationship of the first segmentwith one or more of the other segments; applying the color modificationto at least one other segment having the indicated relationship with thefirst segment.